Q learning with finite trials

The standard reinforcement learningmodel is powerful enough to deal with never ending trials. By slightly discounting rewards obtained in the future, an infinite walk in the environment is still guaranteed to have a finite expected future reward. This however comes at a price. The discounting may corrupt estimates of the expected return in ending trials. Also in most cases algorithms that can deal with continuing trials rely heavily on arbitrary initial estimates of the optimal policy Q . If these estimates are off, convergence can be slowed down considerably. In terminal states the expected future return is 0, this can be known exactly after the first visit. so we can expect we can improve the Q learning algorithm by anticipating on terminal states. This thesis investigates MDPs in which discounting is unnecessary. New work is presented for the deterministic case. In deterministic MDPs, the set of undiscounted Bellman equations for Q has a unique solution if and only if from every state there is a path to a terminal state, and if all cycles have negative weight. In this set of MDPs the Q learning algorithm can be optimized by altering it in such a way that the initial estimates are not used. If the agent trains and explores in a fair way, that is, if the agent never fully ignores a state action pair, this improved Q learning algorithm is proven to converge in finite time. The proof gives a worst case guarantee of the performance. To gain insight in the average case performance, and the effect of the exploration strategy, several experiments in the simple three-in-a-row game of tic-tac-toe have been conducted. New performance measures, based on the (precomputed) optimal table Q are introduced to more accurately study the behavior of the algorithm.